Amazon and Meta appear to be the two companies with the most prolific AI bots when it comes to the scraping of websites worldwide.
Data and technology company 51Degrees says it has analysed data looking at three billion website visits in the year to 29 May 2026 based on sites it works with in the telecoms, advertising, media, retail, financial services and technology sectors.
It found that AmazonBot accounted for more than 4% of all web visits over the period monitored and Meta-External-Agent represented almost 3%.
Publishers have become increasingly concerned about unwanted website scraping which sees their content taken and repurposed without permission and adds to their server costs.
[Read more: Eight in ten of world’s biggest news websites now block AI training bots]
AmazonBot tends to make “broad, high-volume sweeps that fetch far more pages per visit than a search crawler”, according to bot traffic analytics platform Known Agents.
On 29 May, AmazonBot accounted for 6.43 million website sessions monitored by 51Degrees, while Meta-External-Agent accounted for 4.23 million.
Together the two AI bots represented almost two-thirds – 63% – of sessions generated by the 72 AI-related bots identified over the year by 51Degrees.
The research was confined to AI-related bots so did not include the main Google bot used to generate search engine results.
The next two most active AI bots were AhrefsBot and Microsoft’s BingBot, which accounted for 2.17 million visits and 1.62 million visits respectively on 29 May.
AhrefsBot, a web crawler that powers Ahrefs marketing intelligence platform and Yep search engine, represented 2.3% of total web sessions.
BingBot, used by Microsoft’s search engine Bing to discover and index pages for search results, represented 1.3%.
AmazonBot, Meta-External-Agent, AhrefsBot and BingBot together accounted for 85% of the crawl sessions generated by the 72 bots identified.
Anthropic’s ClaudeBot and OpenAI Search Bot ranked seventh and ninth in total web sessions by AI bots on 29 May, generating around 288,000 and 129,000 sessions respectively.
Almost two-thirds rise in AI bot traffic
During the 12 months to the end of May websites monitored by 51Degrees saw a 62.5% increase in bot traffic from AI companies.
AI bots accounted for 13% of web sessions in May 2026, up from 8% a year earlier.
Nearly half of all web traffic is estimated to come from bots, according to infrastructure company Fastly.
James Rosewell, CEO of 51Degrees, said the increase in AI bot traffic to websites is a “huge licensing revenue opportunity” for publishers, “but left unchecked is a growing and unwanted burden, particularly for smaller, independent players who can ill afford the impact of IP theft and increased bandwidth costs.
“Whilst AI bots only represent a small percentage of overall bot traffic they are the most insidious because they not only consume vast quantities of bandwidth but they also use that stolen content to create AI answers that steal traffic from the original creator of that content.”
Rosewell added that while there is “no technical solution that can stop the bots entirely”, 51Degrees has developed a bot protection and payment product for publishers that charges AI bots for unwanted data harvesting.
Publishers blocking one-fifth of AI bots
Robots.txt, the Robots Exclusion Protocol, is designed to stop automated software from crawling sites (as opposed to human visitors) by telling bots which part of a website they should not access.
However, publishers are only blocking around 21% of AI scraper bots in their robots.txt, according to Known Agents.
The bot traffic analytics platform found that publishers Adweek, The San Diego Tribune and the Arkansas Democrat Gazette are currently among those with 100% coverage of AI bots tracked by the company.
W magazine, Screenrant, Semafor, CBS News and Digital Spy were among those with the lowest coverage blocking 2% of bots in their Robots.text files.
Known Agents claims that AI scraper and data providers abide by robots.txt requests 95.6% of the time and that the main issue is that publishers are not being comprehensive in terms of adding bots to their Robots.txt files.
Publishers have raised concerns that AI companies can use third-party companies to scrape content instead.
Email pged@pressgazette.co.uk to point out mistakes, provide story tips or send in a letter for publication on our "Letters Page" blog
